Executing alternative plans for a SQL statement

ABSTRACT

Under automated alternate plan analysis, a query optimizer generates candidate execution plans. The candidate execution plans are selected as alternate execution plans for the query and execution. Output describing characteristics of each alternate execution plan and/or its execution is generated and/or compared. From this information, it may be determined, for example, whether results returned by any of the alternate execution plans are the same and whether the least cost execution plan is actually the most efficiently executed.

RELATED APPLICATIONS

The present application claims priority to U.S. Provisional ApplicationNo. 60/817,998, entitled Executing Alternative Plans For A SQLStatement, filed on Jun. 30, 2006 by Mohamed Zait, the entire content ofwhich is hereby incorporated by reference for all purposes as if fullyset forth herein.

FIELD OF THE INVENTION

The present invention relates to database systems, and in particular, tooptimization of queries executed by a database system.

BACKGROUND

Relational and object-relational database management systems storeinformation in tables of rows. To retrieve data, queries that requestdata are submitted to a database server, which computes the queries andreturns the data requested.

Queries submitted to the database server must conform to the syntacticalrules of a particular query language. One popular query language, knownas the Structured Query Language (SQL), provides users a variety of waysto specify information to be retrieved.

Queries submitted to a database server are evaluated by a queryoptimizer. Based on the evaluation, the query optimizer generates anexecution plan that defines steps for executing the query. Typically,the query optimizer generates an execution plan optimized for efficientexecution.

Several problems can arise with execution plans generated by queryoptimizers. First, the execution plans may not generate the correct theresults. Second, an optimizer execution plan may in fact perform poorly.

Determining the root cause of problems with execution plans can be verycomplicated and tedious. In fact, even detecting such problems can bedifficult.

Based on the foregoing, there is a clear need for developing techniquesthat facilitate detecting and solving problems with execution plans.

The approaches described in this section are approaches that could bepursued, but not necessarily approaches that have been previouslyconceived or pursued. Therefore, unless otherwise indicated, it shouldnot be assumed that any of the approaches described in this sectionqualify as prior art merely by virtue of their inclusion in thissection.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1 is a diagram of a query optimizer according to an embodiment ofthe present invention.

FIG. 2 depicts a procedure for automatically performing alternate queryanalysis according to an embodiment of the present invention.

FIG. 3 is a diagram of a computer system according to an embodiment ofthe present invention.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however,that the present invention may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to avoid unnecessarily obscuring thepresent invention.

When a query optimizer evaluates a query statement, it determinesvarious “candidate execution plans” and estimates a query execution cost(“estimated query cost”) for each. The candidate execution plan with thelowest estimated query cost is assumed to be most efficient and thus isselected by the query optimizer for execution.

The query may be transformed into one or more transformed queries. Forthe query and one or more of the transformed queries, various candidateexecution plans are generated, each, for example, using different joinand access operations and/or performing the operations in a differentorder.

Various problems may be encountered during query optimization.Structures that represent the query may be corrupted when manipulated totransform a query. Query transformations and/or the execution plans maynot return the correct results, and the execution plan with the leastcost may not in fact be the most efficient execution plan.

To facilitate resolving such problems, an automated solution referred toas alternate plan analysis, generates information useful to resolvingthese problems. Under this approach, for a given query, a subset ofcandidate execution plans generated by a query optimizer are selected asalternate execution plans for the query and executed. Output describingcharacteristics of each alternate execution plan and/or its execution isgenerated for comparison. From this information, it may be determined,for example, whether results returned by any of the alternate executionplans are the same and whether the least cost execution plan is actuallythe most efficiently executed.

Illustrative Operational Environment

FIG. 1 is a diagram depicting a query optimizer and related componentswithin a database server (not shown). Generally, a server, such as adatabase server, is a combination of integrated software components andan allocation of computational resources, such as memory, a node, andprocesses on the node for executing the integrated software components,where the combination of the software and computational resources arededicated to providing a particular type of function on behalf ofclients of the server. A database server governs and facilitates accessto a particular database, processing requests by clients to access thedatabase.

A database comprises data and metadata that is stored on a persistentmemory mechanism, such as a set of hard disks. Such data and metadatamay be stored in a database logically, for example, according torelational and/or object-relational database constructs. Databaseapplications interact with a database server by submitting to thedatabase server commands that cause the database server to performoperations on data stored in a database. A database command may be inthe form of a database statement. For the database server to process thedatabase statements, the database statements must conform to a databaselanguage supported by the database server. One non-limiting databaselanguage supported by many database servers is SQL, includingproprietary forms of SQL supported by such database servers as Oracle,(e.g. Oracle Database 10 g). SQL data definition language (“DDL”)instructions are issued to a database server to create or configuredatabase objects, such as tables, views, or complex types.

Generally, data is stored in a database in one or more data containers,each container contains records, and the data within each record isorganized into one or more fields. In relational database systems, thedata containers are typically referred to as tables, the records arereferred to as rows, and the fields are referred to as columns. Inobject oriented databases, the data containers are typically referred toas object classes, the records are referred to as objects, and thefields are referred to as attributes. Other database architectures mayuse other terminology. Systems that implement the present invention arenot limited to any particular type of data container or databasearchitecture. However, for the purpose of explanation, the examples andthe terminology used herein shall be that typically associated withrelational or object-relational databases. Thus, the terms “table”,“row” and “column” shall be used herein to refer respectively to thedata container, record, and field.

Query Optimizer and Execution Plans

Referring to FIG. 1, query parser 110 receives a query statement andgenerates an internal query representation 112 of the query statement.Typically, the internal query representation is a set of interlinkeddata structures that represent various components and structures of aquery statement. The internal query representation may be in the form ofa graph of nodes, each interlinked data structure corresponding to anode and to a component of the represented query statement. The internalrepresentation is typically generated in memory for evaluation,manipulation, and transformation by query optimizer 120.

Query optimizer 120 generates one or more different candidate executionplans for a query, which are evaluated by query optimizer 120 todetermine which should be used to compute the query. The one or morecandidate execution plans that are evaluated for this purpose arecollectively referred to as the plan search space or search space. For agiven query, a search space may include candidate execution plans P₁, P₂through P_(N).

Execution plans may be represented by a graph of interlinked nodes,referred to herein as operators, that each correspond to a step of anexecution plan, referred to herein as an execution plan operation. Thehierarchy of the graphs represents the order in which the execution planoperations are performed and how data flows between each of theexecution plan operations. Execution plan operations include, forexample, a table scan, an index scan, hash-join, sort-merge join,nested-loop join, and filter.

To evaluate the candidate execution plans in the search space, queryoptimizer 120 estimates a cost of each candidate execution plan andcompares the estimated query costs to select an execution plan forexecution. In an embodiment, the estimated query cost is generated by aquery cost estimator 130, which may be a component of query optimizer120. For a plan P_(i) supplied by query optimizer 120, cost estimator130 computes and generates an estimated query cost E_(i). In general,the estimated query cost represents an estimate of computer resourcesexpended to execute an execution plan. The estimated cost may berepresented as the execution time required to execute an execution plan.

Estimating query cost can be very complex. For example, to generate anestimated query cost, query cost estimator 130 may estimate cardinality(the number of rows to scan and process), selectivity (the fraction ofrows from a row set filtered by a predicate), and cost in terms ofresources such as disk input and output, CPU usage, and memory usage ofvarious execution plan operations. The accuracy of these estimatesdepends on statistic about tables (e.g. histograms) as well otherstatistics.

To determine which candidate execution plan in the search space toexecute, query optimizer 120 selects the candidate execution plan withthe lowest estimated cost. To perform alternate plan analysis, queryoptimizer 120 may select multiple candidates for execution. Each of theselected candidates is referred to herein as an alternate executionplan.

Query optimizer 120 may optimize a query by transforming the query. Ingeneral, transforming a query involves rewriting a query into anotherquery that should produce the same result and that can potentially beexecuted more efficiently, i.e. one for which a potentially moreefficient and less costly execution plan can be generated. Examples ofquery transformation include view merging, subquery unnesting, predicatemove-around and pushdown, common subexpression elimination,outer-to-inner join conversion, materialized view rewrite, and startransformation.

The query as transformed is referred to herein as the transformed query.The query is rewritten by manipulating a copy of the queryrepresentation to form a transformed query representation representing atransformed query.

One or more alternate transformations may be performed, and for eachalternate transformation, one or more candidate execution plans aregenerated. Thus, a search space may contain candidate execution plansfor multiple transformations, and multiple candidate execution plans fora single query transformation.

Alternate Plan Analysis Overview

FIG. 2 is a flowchart depicting a procedure for performing alternatequery plan analysis, according to an embodiment of the presentinvention. The procedure is executed for each query in a set of one ormore queries referred to herein as the working query set. The workingquery set can be provided as input by an end user.

According to an embodiment, the procedure is controlled by “analysisparameters.” Analysis parameters describe how alternate plan analysis isperformed. For example, the analysis parameters may specify rankingcriteria for selecting the alternate execution plans from among thecandidate execution plans and how many of the top ranked candidateexecution plans to select as alternate execution plans for alternatequery analysis. The analysis parameters may govern what kind ofinformation to generate about the execution of the alternate executionplans. A type of analysis parameter, referred to herein as an optimizerparameter, specifies the optimization behavior query optimizer 120should follow. Optimization behavior refers to the way a query optimizeroptimizes, e.g. what types of transforms to make or not make, what typesof execution plan operations to use or not use, and what order andorganization of execution plan operators to use or not use.

Analysis parameters may be provided as input from a user. This allowsusers to control and tailor the operation of alternate query analysis.

The steps depicted in FIG. 2 are performed iteratively for each query ina working query set. The particularly query in the working set for whichthe iteration is being performed is referred to herein as the currentquery.

At steps 205, a search space is generated according to the analysisparameters. For example, if the analysis parameters include an optimizerparameter that specifies not to perform specific query transformations,then the query optimizer foregoes the transformations and the searchspace does not include candidate execution plans for such transformedqueries. If the optimizer parameters include an optimizer parameter toexclude certain operators in execution plans, then the query optimizerforegoes generating candidate execution plans using such operators.

At step 210, alternate execution plans are selected from among thesearch space according to the criteria specified by the analysisparameters. For example, an analysis parameter may specify to select thefive candidate execution plans with the lowest estimated costs. Thealternate execution plans selected for execution are referred to hereinas the alternate plan set.

At step 215, the alternate execution plans in the alternate plan set areexecuted and analysis output about the execution is generated. Analysisoutput can include information describing characteristics of analternate execution plan and its execution, such as the performancerealized during execution and whether any errors were encountered, anddetails about the execution plan operations in the alternate executionplans and any query transformations upon which the alternate executionplan is based.

Top-Ranked Comparison

A mode of operation for alternate plan analysis is referred to herein astop-ranked comparison. In one example of top-ranked comparison, thecandidate execution plans with the top N, lowest estimated query costsare selected for the alternate plan set. This mode may be specified byan analysis parameter. The analysis output for a given query may showthe performance realized for each alternate execution plan and itsestimated cost. Such information shows how strongly the estimated querycosts correlate to realized performance and how well the optimizerselects the optimal plan.

For example, for a given query in the working set, where N equals 10, 10alternate execution plans P1 . . . P10 are selected. The analysis outputgenerated for the alternate plan set is shown below.

TABLE AR1 PLAN Cost Estimate Actual Execution Time P1 2.0 2.5 P2 1.5 .7P3 1.8 1.7 P4 1.0 .9 P5 2.3 2.7 . . . . . . . . .

The estimated query costs and actual execution times of the plans notshown above, i.e. plans P6 through P10, are higher than any of thoseshown above for P1 through P5. According to the above table AR1, basedon estimated query costs, a query optimizer would select plan P4,because the estimated cost is 1.0 seconds. However, the fastest planexecuted was plan P2. Although P2 had a cost estimate of 1.5 seconds,which is greater than that of P4's, P2's actual execution time is 0.7seconds, less than the 0.9 second actual execution time of plan P4.Nevertheless, the analysis output shows that the alternate executionplan with the lowest cost estimate was one of the top two performingalternate execution plans.

Analysis output for the whole working set provides a more overallindication of the performance of the query optimizer. For example, areview of the analysis output for the whole working set shows that queryoptimizer selects from among the alternate execution plans in thealternate plan sets, the top two actual performing alternate executionplans 50% of the time, and the top five 75% of the time.

More generally, the ranking criteria of the top-rank mode may be basedon other metrics other than estimated query cost. Other metrics include,for example, estimated memory usage or a number of joins of a certaintype called for by a candidate execution plan. For example, if theranking criteria were based on memory usage, then the plans using thetop ranked amount of memory (i.e. lowest amount) may be selected. Themetric upon which ranking criteria is based can be specified by ananalysis parameter. Also, analysis parameters may specify N as aconstant or a percentage.

In another embodiment, one or more randomly selected alternate executionplans may also be included in the alternate plan set.

Result Set Comparison Mode

In an embodiment, the result sets computed for each alternate executionplan in an alternate plan set are compared to determine whether theresult sets are equal. Unequal result sets indicate that at least onealternate execution plan is not computing the query results correctly.Incorrect results are often a symptom of a query transformation problem.The result set comparison mode may be controlled by an analysisparameter.

The equality or inequality of result sets may be determined bygenerating a checksum for the result set or generating a hash value byapplying a hash function to the result set. Different checksums or hashvalues indicate different result sets. The analysis output may indicatewhich alternate computed the same results and which computed differentresults.

Version Mode

Like software products in general, software that implements queryoptimization and related functions evolves between versions. Eachversion may implement different ways of transforming queries, generatingsearch spaces, and estimating query cost. According to an embodiment,query optimizer 120 is configured to operate as it did at a particularversion. As a result, alternate query analysis can generate and comparealternate execution plans generated for different versions. This abilityfacilitates detecting what version of a query optimizer may haveintroduced errors or inefficiencies.

For example, software for query optimization has evolved through fiveversions. A query ran efficiently in an earlier version. For the query,the analysis parameters may be set to generate an alternate executionplan for each version. The analysis output may identify the alternateexecution plan generated for each version and its actual execution time,revealing, for example, that for the version were execution timedegraded, a different alternate execution plan was selected by the queryoptimizer as the one with the lowest cost, and that the new selectedalternate execution plan implemented a query transformation introducedin that version.

An optimizer parameter may specify to generate alternate execution plansfor specific versions or for all versions between a range of versions.

Finer Grained Control of Optimization Behavior

The version mode represents a coarser-grained way of controllingoptimization behavior. According to an embodiment, optimization featuresmay be controlled at a finer level of granularity. The use of certaintransformations or execution plan operations may be controlled by aspecific parameter. For example, an optimizer parameter may specify thata certain transformation should be enabled, not enabled, or should notbe used under certain conditions.

Conditional Analysis Output

According to an embodiment, the content of analysis output depends onconditions detected and/or results generated during alternate queryanalysis. For example, alternate query analysis is performed to comparethe least costly alternate execution plan generated each version in arange of versions. An analysis parameter may specify that if for anyquery in the working query set the performance of an alternate executionplan for the most recent version has degraded, then the analysis outputfor the query should include information about what alternate executionplans were selected, what transformations were performed, the estimatedquery costs and actual execution time, and other information useful todiagnose the reason underlying the degraded performance.

As another example, alternate query analysis is performed to determinethe top-ranked alternate execution plans. If under result comparison adifference in the computed results of alternate execution plans for aquery is detected, then the analysis output includes more detailedinformation for the alternate execution plans of the query.

Automated Query Analysis Tool

According to an embodiment of the present invention, a software tool,separate and apart from the query optimizer, controls alternate queryanalysis. The query analysis tool reads the analysis parameter as inputand performs the alternate query analysis accordingly, interacting witha query optimizer to control optimization behavior and how the queryoptimizer selects alternate execution plans from the search space toreturn to the tool for execution.

To control optimization behavior, the tool may embed optimizer hintswithin queries that the tool submits to the query optimizer. Optimizerhints are commands that may be embedded within a query statement tospecify to a query optimizer what optimizations to perform or notperform e.g. what execution plan operations to use or not use and whatquery transforms to perform or not perform. The tool may also controloptimization behavior by changing the query compilation environment.

The tool executes each of the alternate execution plans returned by thequery optimizer and generates analysis output based on the results ofthe execution.

The query analysis tool automates many tasks that would be extremelyonerous to perform manually. Regression testing, for example, is an areathat would benefit enormously from the tool. Under regression testing,queries issued by an application to a database server may be tested todetermine whether they run correctly or as efficiently as previously. Anapplication could have thousands and thousands of such queries.

To perform the regression testing manually, a user may manually submitthe queries to both the old and new database servers for execution andcompare the results and performance of all these query executions. Underapproaches described herein, the queries of an application are used toform a working set. Next, alternate query analysis is run under theversion mode and result comparison mode to automatically detect querieswith degraded performance or queries that are producing differentresults.

The task of manually generating and running alternate execution plansfor comparison is also an onerous task to perform manually. Often, acombination of optimizer hints is needed to create a particularexecution plan for a query to force or prevent particular querytransformations. Determining the combination of optimizer hints neededand embedding them within a query statement can be very complex, a taskfurther compounded by having to do this for all alternate executionplans needed for comparison purposes.

Hardwire Overview

FIG. 3 is a block diagram that illustrates a computer system 300 uponwhich an embodiment of the invention may be implemented. Computer system300 includes a bus 302 or other communication mechanism forcommunicating information, and a processor 304 coupled with bus 302 forprocessing information. Computer system 300 also includes a main memory306, such as a random access memory (RAM) or other dynamic storagedevice, coupled to bus 302 for storing information and instructions tobe executed by processor 304. Main memory 306 also may be used forstoring temporary variables or other intermediate information duringexecution of instructions to be executed by processor 304. Computersystem 300 further includes a read only memory (ROM) 308 or other staticstorage device coupled to bus 302 for storing static information andinstructions for processor 304. A storage device 310, such as a magneticdisk or optical disk, is provided and coupled to bus 302 for storinginformation and instructions.

Computer system 300 may be coupled via bus 302 to a display 312, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 314, including alphanumeric and other keys, is coupledto bus 302 for communicating information and command selections toprocessor 304. Another type of user input device is cursor control 316,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 304 and forcontrolling cursor movement on display 312. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

The invention is related to the use of computer system 300 forimplementing the techniques described herein. According to oneembodiment of the invention, those techniques are performed by computersystem 300 in response to processor 304 executing one or more sequencesof one or more instructions contained in main memory 306. Suchinstructions may be read into main memory 306 from anothermachine-readable medium, such as storage device 310. Execution of thesequences of instructions contained in main memory 306 causes processor304 to perform the process steps described herein. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions to implement the invention. Thus,embodiments of the invention are not limited to any specific combinationof hardware circuitry and software.

The term “machine-readable medium” as used herein refers to any mediumthat participates in providing data that causes a machine to operationin a specific fashion. In an embodiment implemented using computersystem 300, various machine-readable media are involved, for example, inproviding instructions to processor 304 for execution. Such a medium maytake many forms, including but not limited to, non-volatile media, andvolatile media. Non-volatile media includes, for example, optical ormagnetic disks, such as storage device 310. Volatile media includesdynamic memory, such as main memory 306. All such media must be tangibleto enable the instructions carried by the media to be detected by aphysical mechanism that reads the instructions into a machine.

Common forms of machine-readable media include, for example, a floppydisk, a flexible disk, hard disk, magnetic tape, or any other magneticmedium, a CD-ROM, any other optical medium, punchcards, papertape, anyother physical medium with patterns of holes, a RAM, a PROM, and EPROM,a FLASH-EPROM, any other memory chip or cartridge, or any other mediumfrom which a computer can read.

Various forms of machine-readable media may be involved in carrying oneor more sequences of one or more instructions to processor 304 forexecution. For example, the instructions may initially be carried on amagnetic disk of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 300 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 302. Bus 302 carries the data tomain memory 306, from which processor 304 retrieves and executes theinstructions. The instructions received by main memory 306 mayoptionally be stored on storage device 310 either before or afterexecution by processor 304.

Computer system 300 also includes a communication interface 318 coupledto bus 302. Communication interface 318 provides a two-way datacommunication coupling to a network link 320 that is connected to alocal network 322. For example, communication interface 318 may be anintegrated services digital network (ISDN) card or a modem to provide adata communication connection to a corresponding type of telephone line.As another example, communication interface 318 may be a local areanetwork (LAN) card to provide a data communication connection to acompatible LAN. Wireless links may also be implemented. In any suchimplementation, communication interface 318 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

Network link 320 typically provides data communication through one ormore networks to other data devices. For example, network link 320 mayprovide a connection through local network 322 to a host computer 324 orto data equipment operated by an Internet Service Provider (ISP) 326.ISP 326 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 328. Local network 322 and Internet 328 both use electrical,electromagnetic or optical signals that carry digital data streams.

Computer system 300 can send messages and receive data, includingprogram code, through the network(s), network link 320 and communicationinterface 318. In the Internet example, a server 330 might transmit arequested code for an application program through Internet 328, ISP 326,local network 322 and communication interface 318.

The received code may be executed by processor 304 as it is received,and/or stored in storage device 310, or other non-volatile storage forlater execution.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. Thus, the sole and exclusive indicatorof what is the invention, and is intended by the applicants to be theinvention, is the set of claims that issue from this application, in thespecific form in which such claims issue, including any subsequentcorrection. Any definitions expressly set forth herein for termscontained in such claims shall govern the meaning of such terms as usedin the claims. Hence, no limitation, element, property, feature,advantage or attribute that is not expressly recited in a claim shouldlimit the scope of such claim in any way. The specification and drawingsare, accordingly, to be regarded in an illustrative rather than arestrictive sense.

1. A method of generating output for analyzing how well a query optimizer generates alternate execution plans and selects an optimal execution plan, the method comprising: receiving a query; in response to receiving the query, automatically performing: generating at least two alternate execution plans for the received query, wherein each alternate execution plan of the at least two alternate execution plans defines operations for executing said received query differently from each other alternate execution plan of the at least two alternate execution plans; for each execution plan of the at least two alternate execution plans, causing execution of said received query by a database server that performs the operations defined by the execution plan; wherein the step of causing execution of said received query for each execution plan of the at least two alternate execution plans causes at least two executions of the received query; and generating and storing output that describes characteristics of each of the at least two executions of said received query that is executed according to the operations defined by the execution plan of the at least two alternate execution plans; wherein the output is useable for analyzing how well the query optimizer generates alternate execution plans and selects the optimal execution plan; wherein the method is performed by one or more computing devices.
 2. The method of claim 1, wherein: said step of generating at least two alternate execution plans is based on a set of parameters; said set of parameters specify one or more versions of said query optimizer upon which an alternate execution plan of said at least two execution plans should be based; and wherein generating said at least two alternate execution plans includes generating for each version of said one or more versions an alternate execution plan based on optimization behavior associated with said each version.
 3. The method of claim 1, wherein the step of generating at least two alternate execution plans includes: generating a search space of a plurality of candidate execution plans; and selecting from the search space said at least two alternate execution plans.
 4. The method of claim 3, wherein the step of selecting is based on an estimated query cost computed from a set of parameters.
 5. The method of claim 3, further comprising: ranking the plurality of candidate execution plans based on at least one of the following metrics: estimated query cost, estimated CPU usage, estimated memory usage, and estimated disk input and output.
 6. The method of claim 5, wherein said step of selecting said at least two alternate execution plans comprises selecting a set of top ranked candidate execution plans based on said step of ranking the plurality of candidate execution plans.
 7. The method of claim 3, wherein said step of selecting said at least two alternate execution plans comprises randomly selecting at least one of said plurality of candidate execution plans.
 8. The method of claim 1, wherein for the at least two alternate execution plans, said output indicates at least one of the following characteristics: a performance realized for each alternate execution plan of said at least two alternate execution plans, an indication of whether a first result returned for one of said at least two alternate execution plans differs from a second result for another of said at least two alternate execution plans, or an indication that an execution error was encountered during execution of said at least two alternate execution plans.
 9. The method of claim 1, wherein said step of generating at least two alternate execution plans is based on a set of parameters that specify how to form execution plans.
 10. The method of claim 9, wherein: the set of parameters specify certain one or more execution plan operations to exclude; and wherein the step of generating at least two alternate execution plans includes foregoing generation of execution plans that include said certain one or more execution plan operations.
 11. The method of claim 9, wherein: the set of parameters specify certain one or more execution plan operations to include; and wherein the step of generating at least two alternate execution plans includes limiting generation of alternate execution plans to execution plans that include said certain one or more execution plan operations.
 12. The method of claim 9, wherein: the set of parameters specify at least one query transformation not to perform; and the steps further include foregoing performing one or more query transformations of said each query that do not include said at least one query transformation.
 13. The method of claim 9, wherein: the set of parameters specify to perform at least one query transformation; and the steps further include performing one or more query transformations in response to detecting that said set of parameters specify to perform said at least one query transformation.
 14. The method of claim 1, further comprising, based at least in part on the output, analyzing whether the optimal execution plan selected by the optimizer was more efficiently executed than other alternate execution plans not selected by the optimizer.
 15. A non-transitory computer-readable storage medium that stores instructions which, when executed by one or more processors, cause the one of more processors to generate output for analyzing how well a query optimizer generates alternate execution plans and selects an optimal execution plan by causing the one or more processors to perform: receiving a query; in response to receiving the query, automatically performing: generating at least two alternate execution plans for the received query, wherein each alternate execution plan of the at least two alternate execution plans defines operations for executing said received query differently from each other alternate execution plan of the at least two alternate execution plans; for each execution plan of the at least two alternate execution plans, causing execution of said received query by a database server that performs the operations defined by the execution plan; wherein the step of causing execution of said received query for each execution plan of the at least two alternate execution plans causes at least two executions of the received query; and generating and storing output that describes characteristics of each of the at least two executions of said received query that is executed according to the operations defined by the execution plan of the at least two alternate execution plans; wherein the output is useable for analyzing how well the query optimizer generates alternate execution plans and selects the optimal execution plan.
 16. The computer-readable storage medium of claim 15, wherein the instructions, when executed by the one or more processors, cause the one of more processors to perform said step of generating at least two alternate execution plans based on a set of parameters; wherein said set of parameters specify one or more versions of said query optimizer upon which an alternate execution plan of said at least two execution plans should be based; and wherein the instructions, when executed by the one or more processors, cause the one of more processors to perform said step of generating said at least two alternate execution plans by generating for each version of said one or more versions an alternate execution plan based on optimization behavior associated with said each version.
 17. The computer-readable storage medium of claim 15, wherein the instructions, when executed by the one or more processors, cause the one of more processors to perform the step of generating at least two alternate execution plans by: generating a search space of a plurality of candidate execution plans; and selecting from the search space said at least two alternate execution plans.
 18. The computer-readable storage medium of claim 17, wherein the instructions, when executed by the one or more processors, cause the one of more processors to perform the step of selecting based on an estimated query cost computed from a set of parameters.
 19. The computer-readable storage medium of claim 17, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform: ranking the plurality of candidate execution plans based on at least one of the following metrics: estimated query cost, estimated CPU usage, estimated memory usage, and estimated disk input and output.
 20. The computer-readable storage medium of claim 19, wherein the instructions, when executed by the one or more processors, cause the one of more processors to further perform said step of selecting said at least two alternate execution plans by selecting a set of top ranked candidate execution plans based on said step of ranking the plurality of candidate execution plans.
 21. The computer-readable storage medium of claim 17, wherein the instructions, when executed by the one or more processors, cause the one of more processors to further perform said step of selecting said at least two alternate execution plans by randomly selecting at least one of said candidate execution plans.
 22. The computer-readable storage medium of claim 15, wherein for the at least two alternate execution plans, said output indicates at least one of the following characteristics: a performance realized for each alternate execution plan of said at least two alternate execution plans, an indication of whether a first result returned for one of said at least two alternate execution plans differs from a second result for another of said at least two alternate execution plans, or an indication that an execution error was encountered during execution of said at least two alternate execution plans.
 23. The computer-readable storage medium of claim 15, wherein the instructions, when executed by the one or more processors, cause the one of more processors to perform said step of generating at least two alternate execution plans based on a set of parameters that specify how to form execution plans.
 24. The computer-readable storage medium of claim 23, wherein: the set of parameters specify certain one or more execution plan operations to exclude; and wherein the instructions, when executed by the one or more processors, cause the one of more processors to perform the step of generating at least two alternate execution plans by foregoing generation of execution plans that include said certain one or more execution plan operations.
 25. The computer-readable storage medium of claim 23, wherein: the set of parameters specify certain one or more execution plan operations to include; and wherein the instructions, when executed by the one or more processors, cause the one of more processors to perform the step of generating at least two alternate execution plans by limiting generation of alternate execution plans to execution plans that include said certain one or more execution plan operations.
 26. The computer-readable storage medium of claim 15, wherein: the set of parameters specify at least one query transformation not to perform; and the instructions, when executed by the one or more processors, cause the one of more processors to further perform foregoing performance of one or more query transformations of said each query that do not include said at least one query transformation.
 27. The computer-readable storage medium of claim 15, wherein: the set of parameters specify to perform at least one query transformation; and the instructions, when executed by the one or more processors, cause the one of more processors to further perform one or more query transformations in response to detecting that said set of parameters specify to perform said at least one query transformation.
 28. The computer-readable storage medium of claim 15, wherein the instructions, when executed by the one or more processors, cause the one of more processors to further perform analyzing whether the optimal execution plan selected by the optimizer was more efficiently executed than other alternate execution plans not selected by the optimizer. 